A topological data analysis library.
Core algorithm written in Scala, using Apache Spark.
Executed in a Jupyter notebook, using the Apache Toree kernel and declarative widgets.
Graphs rendered with Sigma/Linkurious, wrapped in a Polymer component.
In [1]:
%AddDeps org.apache.spark spark-mllib_2.10 1.6.2
%AddDeps org.scalanlp breeze-natives_2.10 0.12
%AddDeps com.github.haifengl smile-core 1.1.0 --transitive
%AddDeps io.reactivex rxscala_2.10 0.26.1 --transitive
%AddDeps com.softwaremill.quicklens quicklens_2.10 1.4.4
%AddDeps com.chuusai shapeless_2.10 2.3.0 --repository https://oss.sonatype.org/content/repositories/releases/
%AddDeps org.tmoerman plongeur-spark_2.10 0.3.15 --repository file:/Users/tmo/.m2/repository
In [2]:
%addjar http://localhost:8888/nbextensions/declarativewidgets/declarativewidgets.jar
In [3]:
import rx.lang.scala.{Observer, Subscription, Observable}
import rx.lang.scala.subjects.PublishSubject
import rx.lang.scala.subjects._
import shapeless.HNil
import org.tmoerman.plongeur.tda._
import org.tmoerman.plongeur.tda.Model._
import org.tmoerman.plongeur.tda.cluster.Clustering._
import org.tmoerman.plongeur.tda.cluster.Scale._
import declarativewidgets._
initWidgets
import declarativewidgets.WidgetChannels.channel
In [4]:
import java.util.concurrent.atomic.AtomicReference
case class SubRef(val ref: AtomicReference[Option[Subscription]] = new AtomicReference[Option[Subscription]](None)) extends Serializable {
def update(sub: Subscription): Unit = ref.getAndSet(Option(sub)).foreach(old => old.unsubscribe())
def reset(): Unit = update(null)
}
In [5]:
%%html
<link rel='import' href='urth_components/paper-slider/paper-slider.html'
is='urth-core-import' package='PolymerElements/paper-slider'>
<link rel='import' href='urth_components/paper-button/paper-button.html'
is='urth-core-import' package='PolymerElements/paper-button'>
<link rel='import' href='urth_components/plongeur-graph/plongeur-graph.html'
is='urth-core-import' package='tmoerman/plongeur-graph'>
<link rel='import' href='urth_components/urth-viz-scatter/urth-viz-scatter.html' is='urth-core-import'>
Out[5]:
Keep references to Rx subscriptions apart.
In [6]:
val in$_subRef = SubRef()
Instantiate a PublishSubject. This stream of TDAParams instances represents the input of a TDAMachine. The PublishSubject listens to changes and sets these to the channel "ch_TDA_1" under the "params" key.
TODO: unsubscribe previous on re-evaluation
In [7]:
val in$ = PublishSubject[TDAParams]
in$_subRef.update(in$.subscribe(p => channel("ch_TDA_1").set("params", p.toString)))
Create an initial TDAParams instance. In the same cell, we submit the instance to the PublishSubject.
In [8]:
val tdaParams =
TDAParams(
lens = TDALens(
Filter("PCA" :: 0 :: HNil, 20, 0.6)),
clusteringParams = ClusteringParams(),
scaleSelection = histogram(10))
in$.onNext(tdaParams)
In [9]:
import org.apache.spark.rdd.RDD
import org.apache.commons.lang.StringUtils.trim
import org.apache.spark.mllib.linalg.Vectors.dense
def readData(file: String) =
sc.
textFile(file).
map(_.split(",").map(trim)).
zipWithIndex.
map{ case (Array(x, y, z), idx) => dp(idx, dense(x.toDouble, y.toDouble, z.toDouble))}
In [10]:
val data_path = "/Users/tmo/Work/batiskav/projects/plongeur/scala/plongeur-spark/src/test/resources/data/"
val horse = data_path + "horse.csv"
val rdd = readData(horse).cache
val ctx = TDAContext(sc, rdd)
Turn a TDAResult into a data structure.
In [11]:
val r = scala.util.Random
def format(result: TDAResult) = Map(
"nodes" -> result.clusters.map(c =>
Map(
"id" -> c.id.toString,
"label" -> c.id.toString,
"size" -> c.dataPoints.size,
"x" -> r.nextInt(100),
"y" -> r.nextInt(100))),
"edges" -> result.edges.map(e => {
val (from, to) = e.toArray match {case Array(f, t) => (f, t)}
Map(
"id" -> s"$from--$to",
"source" -> from.toString,
"target" -> to.toString)}))
Run the machine, obtaining an Observable of TDAResult instances
In [12]:
val out$: Observable[(TDAParams, TDAResult)] = TDAMachine.run(ctx, in$)
In [13]:
val out$_subRef = SubRef()
In [14]:
out$_subRef.update(
out$.subscribe(
onNext = (t) => t match {case (p, r) => channel("ch_TDA_1").set("result", format(r))},
onError = (e) => println("Error in TDA machine: ", e)))
In [15]:
val pipe$_subRef = SubRef()
val nrBins$ = PublishSubject[Int]
val overlap$ = PublishSubject[Percentage]
In [16]:
channel("ch_TDA_1").watch("nrBins", (_: Any, v: Int) => nrBins$.onNext(v))
channel("ch_TDA_1").watch("overlap", (_: Any, v: Int) => overlap$.onNext(BigDecimal(v) / 100))
In [23]:
import TDAParams._
val BASE =
TDAParams(
lens = TDALens(
Filter("eccentricity" :: "infinity" :: HNil, 15, 0.5)),
clusteringParams = ClusteringParams(),
scaleSelection = histogram(50),
collapseDuplicateClusters = false)
val params$ =
List(
nrBins$.map(v => setFilterNrBins(0, v)),
overlap$.map(v => setFilterOverlap(0, v))).
reduce(_ merge _).
scan(BASE)((params, fn) => fn(params))
pipe$_subRef.update(params$.subscribe(in$))
channel("ch_TDA_1").set("nrBins", BASE.lens.filters(0).nrBins)
channel("ch_TDA_1").set("overlap", (BASE.lens.filters(0).overlap * 100).toInt)
We create two slider widgets that provide the inputs for the nrBins$ and overlap$ Observables.
In [24]:
%%html
<template is='urth-core-bind' channel='ch_TDA_1'>
<table style="border-style: hidden;">
<tr style="border-style: hidden;">
<th style="border-style: hidden;">nr of bins</th>
<td style="border-style: hidden;">
<paper-slider min="0" max="100" step="1" value="{{nrBins}}"></paper-slider>
</td>
<td style="border-style: hidden;">[[nrBins]]</td>
</tr>
<tr style="border-style: hidden;">
<th style="border-style: hidden;">overlap</th>
<td style="border-style: hidden;">
<paper-slider min="0" max="75" step="1" value="{{overlap}}"></paper-slider>
</td>
<td style="border-style: hidden;">[[overlap]]%</td>
</tr>
</table>
</template>
Out[24]:
In [19]:
%%html
<template is='urth-core-bind' channel='ch_TDA_1'>
<plongeur-graph data="{{result}}"></plongeur-graph>
</template>
Out[19]:
In [20]:
%%html
<template is='urth-core-bind' channel='ch_TDA_1'>
<div style='background: #FFB; padding: 10px;'>
<span style='font-family: "Courier"'>[[params]]</span>
</div>
</template>
Out[20]:
In [32]:
%%html
<template is='urth-core-bind' channel='data'>
<urth-viz-scatter
datarows='[[raw]]'
primary='0'
secondary='1'
/>
</template>
Out[32]:
In [30]:
val rawData = rdd.
map(dp => {
val x = dp.features(2)
val y = dp.features(1)
List(x, y)}).collect.toList
rawData.take(3)
Out[30]:
In [31]:
channel("data").set("raw", rawData)
In [ ]:
In [ ]: